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At least half of states administer or are developing kindergarten entry assessments. In fall 2017 the Illinois 
State Board of Education began requiring teachers to report data on every child’s skills at kindergarten 
entry using the Kindergarten Individual Development Survey. State and local stakeholders have asked for 


more information on the reliability and validity of the survey and on the gaps in children’s skills at school 
entry. This study analyzed the psychometric properties of the 14 required items on the survey after its 
first statewide administration. It examined average skills and the variation in skill levels at kindergarten 
entry, as well as their differences across child subgroups and school poverty levels. And it interviewed 
teachers and principals about barriers in administering the survey and suggestions for improvement. 


The study found that the survey measures two developmental domains: learning and social skills, and 
academic knowledge and skills. Measures of these domains are psychometrically reliable and valid. 
Nearly 9 in 10 children (88 percent) had a score below the scale’s midpoint for the learning and social 
skills domain, and 85 percent had a score below the scale’s midpoint for the academic knowledge and 
skills domain. Children’s skills differed across subgroups. The percentage of children in a school who were 
eligible for the national school lunch program was negatively associated with academic knowledge and 
skills at kindergarten entry, even after child-level eligibility for the program was controlled for. Teachers 
and principals reported multiple challenges in administering the survey—including difficulties observing 
all skills for every child, choosing between adjacent rating categories, and entering data into the online 
portal—and had several suggestions for improvement. 


At least half of states administer or are developing kindergarten entry assessments (Education Northwest, 2016). 
Many states, including Illinois, received federal support to develop kindergarten entry assessments through the 
Race to the Top Early Learning Challenge Grants. The Illinois State Board of Education—in partnership with an 
advisory committee of experts in early childhood education and child develop- 
ment and the WestEd Center for Child and Family Studies—adapted a California 
kindergarten entry assessment to create the Kindergarten Individual Develop- For additional information, 
ment Survey. The state required all kindergarten teachers to use this kindergar- including background 

ten entry assessment statewide for the first time in the 2017/18 school year. on the survey, technical 


methods, and supporting 
analyses, access the report 
appendices at 
https://go.usa.gov/xVtfA. 


The Kindergarten Individual Development Survey is an observational assessment 
performed by teachers. Illinois kindergarten teachers receive a one-day training 
on how to use the survey instrument and enter data into KIDStech, the associated 
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online data portal. Teachers then observe each child during the first 40 days of instruction, interview the child’s 
family or other school staff as necessary, and gather examples of the child’s work. Based on those data, teachers 
select one of six developmental ratings for each of the survey’s 14 items (Illinois State Board of Education, 2015). 
(See box 1 for definitions of key terms used in the report and appendix A for more details on the survey, including 
the items and the process used to develop and pilot it.) 


Box 1. Key terms 


Domain. An area of knowledge within a child’s school readiness—for example, social-emotional development, language and liter- 
acy skills, or math. 


Eligibility for the national school lunch program. Used in this study as a proxy for economic disadvantage. The national school 
lunch program is designed to benefit children whose families have unmet economic needs or children who attend schools with 
high numbers of children whose families have unmet economic needs. 


Eligibility for an individualized education program. An indication of whether a child is eligible to receive special education 


services. 


Factor. A representation of a trait that is not on its own observable or measurable but is reflected by individual items that are 
statistically similar in response patterns. 


Item. An individual question on the Kindergarten Individual Development Survey. 


Rating. One of six categories that teachers report that best represents a child’s developmental stage for an item on the Kin- 
dergarten Individual Development Survey. Though the anchors for the rating categories differ across items, the names of the 
rating categories are the same: building—earlier, building—middle, building—later, integrating—earlier, integrating—middle, and 
integrating—later. 


Reliability. The reproducibility of an estimate. In this study it is the probability that children estimated to have higher skills than 
other children actually do have higher skills. 


Sum-score index. A teacher’s average rating for a domain across all its items. It is calculated by converting the ratings on individu- 
al items to a numerical scale, summing the converted ratings across individual items, and dividing the sum by the number of items 
over which the sum was calculated. The study reports sum-score indexes (rather than scale scores, which are on a continuous 
scale and generated based on statistical models) because of the ease of interpretation. 


Validity. The extent to which the scale or sum-score index measures the domain that it is intended to measure. 


The Kindergarten Individual Development Survey provides new opportunities for data-informed decisionmaking 
for teachers and principals. Teachers have immediate access to summary reports for each child and for the class- 
room. Teachers can use the reports about each child to communicate with parents about what each child knows 
and how to help the child advance. Teachers can use the classroom summary reports to inform the content, 
pacing, and differentiation of their instruction. And for the first time, the results provide data that principals and 
district leaders can use to examine differences in skills and trends in skill development across districts, schools, 
and child subgroups. Differences in skills at kindergarten entry revealed by the data can inform conversations 
between K-12 and early childhood policymakers about how to best prepare children for kindergarten. 


Because of the potential influence of the Kindergarten Individual Development Survey on classroom practice 


and early childhood education policy, its quality and usability are critical to regional stakeholders. The Midwest 
Early Childhood Education Research Alliance requested that the Regional Educational Laboratory (REL) Midwest 


REL 2020-012 2 


explore three issues.’ First, alliance members wanted to confirm the psychometric analysis established during 
the pilot phase (based on data on 26,495 Illinois kindergarteners from the survey’s 2014/15 pilot administration), 
which found that the survey measured three domains (University of California—Berkeley Evaluation and Assess- 
ment Research Center & WestEd Center for Child and Family Studies, 2015). In addition, because the survey offers 
the first opportunity to examine differences in skills at kindergarten entry, alliance members requested analyses 
of gaps in skills across child subgroups and school poverty levels. Finally, alliance members wanted to hear from 
teachers and principals about the barriers they faced in administering the survey and about how to improve it. 
The results of this study will inform REL Midwest’s technical assistance and coaching work with Illinois educators 
on how to use data from the Kindergarten Individual Development Survey. 


Research questions 

The study addressed seven research questions in three categories. 

Questions about what the Kindergarten Individual Development Survey measures and for whom 

1. Does the Kindergarten Individual Development Survey measure the three developmental domains found 
during the 2014/15 pilot administration of the survey (language and literacy development, cognition: mathe- 
matics, and approaches to learning/social-emotional development)? 

2. Are the measures of the developmental domains valid across child subgroups? 


Questions about students’ knowledge and skills 


3. What knowledge and skills do Illinois children have at kindergarten entry, on average, and how do knowledge 
and skills vary within each developmental domain? 


4. Are there differences in children’s knowledge and skills at kindergarten entry across key subgroups (such as 
children eligible for the national school lunch program and those not eligible, boys and girls, English learner 
children and non-English learner children, children eligible for an individualized education program and those 
not eligible, and children of different races/ethnicities)? 

5. Is there an association between the percentage of children in a school who are eligible for the national school 
lunch program and academic knowledge and skills at kindergarten entry? Does the association exist after other 
child-level characteristics are controlled for? 


Questions about educators’ use of the Kindergarten Individual Development Survey 


6. What barriers did teachers and principals encounter in administering the survey in its first statewide 
administration? 


7. What suggestions do teachers and principals have for improving survey administration? 


See box 2 for a summary of the study’s data sources, sample, and methods and appendix B for more details. 


1. The Midwest Early Childhood Education Research Alliance core members represent the following organizations: Brokaw Early Learn- 
ing Center, Elgin Area School District U-46, Illinois Department of Children and Family Services, Illinois Department of Human Services, 
Illinois Department of Public Health, Illinois Governor’s Office of Early Childhood Development, Illinois Head Start Association, Illinois 
State Board of Education, Northern Illinois University, Oswego Area Community Unified School District 308, Rockford Public Schools, and 
Springfield Public Schools. 
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Box 2. Data sources, sample, and methods 


Data sources. The study used data from four sources: 

e Teacher ratings for children on the 14 required items (of 55 total items) from the fall 2017 administration of the Kindergarten 
Individual Development Survey in Illinois provided by the Illinois State Board of Education (see table A1 in appendix A for a list 
of items and their definitions and figure Al for an example of the rating categories as they appear to teachers when they use 
the survey). Teachers rated each child on each item using one of six ratings: building—earlier, building—middle, building—later, 
integrating—earlier, integrating—middle, and integrating—later.' 

¢ Child demographic data from records provided by the Illinois State Board of Education. 

e School-level data on the percentage of children eligible for the national school lunch program from the Common Core of Data 
(U.S. Department of Education, 2019). 

e Interviews conducted with a sample of one kindergarten teacher and one principal from each of nine schools. 


Sample. The data from the fall 2017 administration of the Kindergarten Individual Development Survey included observations on 
125,800 children in 6,275 classrooms in 2,077 public schools in 745 districts. Excluded from the analytic sample were 9,683 chil- 
dren for whom teachers had entered ratings for fewer than 12 of the survey’s 14 items and 2,401 children who had received ratings 
for alternate items because of their English learner status.? That left an analytic sample of 113,716 children in 5,872 classrooms in 
2,039 schools in 738 districts, or about 86 percent of the 2017/18 kindergarten cohort (children entering a publicly funded kinder- 
garten in Illinois in fall 2017). (See table B1 in appendix B for a comparison of the demographic characteristics of children in the 
analytic sample and children in the 2017/18 Illinois kindergarten cohort.) The sample for teacher and principal interviews included 
nine kindergarten teachers and nine principals in Illinois schools (one teacher and one principal from each of nine schools) who 
were nominated by regional Kindergarten Individual Development Survey coaches (who are employees of the Illinois State Board 
of Education). (See table B2 in appendix B for details on the interview sample.) 


Methodology. One objective of the study was to test whether the survey’s 14 required items measure the three developmental 
domains found during the 2014/15 pilot administration of the survey (research question 1). To determine that, the study team 
performed exploratory and confirmatory factor analyses as well as other psychometric analyses. These analyses tested how the 
items on the Kindergarten Individual Development Survey group together to form reliable and valid measures and determined the 
number of domains measured by the survey items and the group of items that measure each domain. Domain structures were 
compared across subgroups to determine whether the measures of the developmental domains are valid across child subgroups 
(research question 2). 


Using the items from each domain, the study team also created a sum-score index to answer research questions 3-5. The 
sum-score index had a .9 correlation with the scale score, so the study team opted to use the sum-score index rather than scale 
scores because of the ease of interpretation. To describe the average knowledge and skills of Illinois children at kindergarten entry 
and the variance of knowledge and skills within each developmental domain (research question 3), the study team calculated the 
frequencies, means, standard deviations, ranges, and distributions of the sum-score index for each domain for the full sample of 
children. 

To determine whether there are differences in knowledge and skills at kindergarten entry across child subgroups (research 
question 4), the study used multilevel modeling. Multilevel modeling is often employed to analyze student data because it 
accounts for the fact that students who are educated within the same schools and districts tend to be more alike in their char- 
acteristics than children chosen at random from the population at large. To help readers understand whether each difference 
(referred to as a skill gap) is substantively meaningful, the sizes of the gaps are reported as effect sizes (Hedges’ g). Furthermore, 
for the academic knowledge and skills domain, the gaps are reported as equivalent number of instruction days (see appendix B 
for information on how the gaps were translated into instruction days). A gap of 18 days or more was considered substantively 
meaningful because research indicates that missing more than 10 percent of instructional time is negatively correlated with later 
student achievement (Romero & Lee, 2007). 

Multilevel modeling was also used to examine the association between the percentage of children in a school who were 
eligible for the national school lunch program and academic knowledge and skills at kindergarten entry (research question 5). (See 
appendix C for supporting analyses for the psychometric and descriptive findings of the report.) 
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To identify barriers and suggestions related to survey administration (research questions 6 and 7), the study team examined 
transcripts from audio recordings of teacher and school administrator interviews. Two study team members reviewed the inter- 
view data individually and coded them separately, looking for themes based on the literature related to assessment in early edu- 
cation (for example, National Research Council, 2008; Snow, 2011) but also allowing for other possible themes to emerge. The two 
study team members then reconciled their findings to establish final, common themes. (See appendix B for more on the methods 
for analyzing interview data.) 

Notes 


1. The ratings fall into two categories: building and integrating. The building category is related to a developmental stage in which the child exhibits 
“Knowledge, skills, or behaviors that demonstrate growing understanding of how people and objects related to one another, how to investigate ideas, 
and how things work. Children use language to express thoughts and feelings, to learn specific early literacy and numeracy skills, and to increasingly 
participate in small group interactions and cooperative activities with others” (Illinois State Board of Education, 2015, p. iv). The integrating category is 
related to a developmental stage in which the child exhibits “Knowledge, skills, or behaviors that demonstrate the increasing ability to connect and com- 
bine strategies in order to express, interpret, and explain complex thoughts and feelings, solve multi-step problems through systematic investigation and 
application of abstract ideas, and participate in a wide range of activities that involve social-emotional, self-regulatory, cognitive, linguistic, and physical 
skills. Children build their capacity to engage in mutually supportive relationships and interactions” (Illinois State Board of Education, 2015, p. iv). 


2. The alternate items were not included in the psychometric analyses because there was no way to equate the alternate items with the rest of the items. 


Findings 
This section presents the main findings. (See appendix C for supporting analyses.) 


Analyses of data from the 2017/18 administration of the Kindergarten Individual Development 
Survey support the presence of two developmental domains: learning and social skills, and academic 
knowledge and skills 


The state currently reports that the 14 required items on the Kindergarten Individual Development Survey 
combine to measure three developmental domains: language and literacy development, cognition: mathemat- 
ics, and approaches to learning/social-emotional development. The choice to report on three developmental 
domains is based on analyses of pilot data from the 2014/15 school year by the University of California—Berkeley 
Evaluation and Assessment Research Center and the WestEd Center for Child and Family Studies. 


By contrast, this study’s analyses of 2017/18 data from the statewide administration of the Kindergarten Individ- 
ual Development Survey provides evidence that the 14 items combine to measure two developmental domains: 
learning and social skills, and academic knowledge and skills (table 1). Evidence supporting the simpler two-domain 
structure for the data is based in part on an examination of the correlations between the factors in the two-domain 
solution versus the correlations between the factors in the three-domain solution. The correlation between the 
learning and social skills factor and the academic knowledge and skills factor is .64, which is high but acceptable 
and indicates that the two domains are closely related but still unique. The correlations between the language and 
literacy development factor and the other two factors in the state-reported three-factor solution were .70. Typical- 
ly, a correlation higher than .70 suggests that factors should be combined (Gaskin, 2016). Additional discussion of 
evidence supporting the simpler two-domain structure for the data is in appendix B. 
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Table 1. Kindergarten Individual Development Survey items, the domains they measure, and associated factor 
loadings 


Factor loadings for domains 
identified in the current study 


Domains identified during the  Learningand Academic knowledge 
2014/15 pilot administration Survey items social skills and skills 
Language and literacy Communication and use of language (expressive) 631 
development Reciprocal communication and conversation 674 
Comprehension of age-appropriate text 498 
Phonological awareness 727 
Letter and word knowledge .849 
Cognition: mathematics Classification of objects into groups based on their attributes .661 
Number sense of quantity .926 
Ability to add and subtract small quantities .858 
Knowledge of shapes 774 
Approaches to learning/social- Relationships and social interactions with familiar adults .877 
emotional development Relationships and social interactions with peers 886 
Curiosity and initiative in learning .652 
Self-control of feelings and behavior .847 
Engagement and persistence 786 


Note: One survey item (comprehension of age-appropriate text) had loadings higher than .4 for both the learning and social skills domain and the academic 
knowledge and skills domain. It was included in the academic knowledge and skills domain because its loading for that domain was larger, indicating that it is 
conceptually more related to that domain. Two items from the language and literacy development domain identified by the University of California—Berkeley 
Evaluation and Assessment Research Center and the WestEd Center for Child and Family Studies (based on pilot data from the 2014/15 school year)—commu- 
nication and use of language (expressive) and reciprocal communication and conversation—had higher loadings for the learning and social skills domain (.631 
and .674, respectively) than for the academic knowledge and skills domain (.326 and .291). The sample consisted of 113,716 children in 2,039 schools. 


Source: Authors’ analyses of fall 2017 Kindergarten Individual Development Survey data provided by the Illinois State Board of Education. 


The analyses found seven items that, taken together, appeared to represent a domain identified as learning and 
social skills. The seven items were related to communication and use of language (expressive), reciprocal com- 
munication and conversation, relationships and social interactions with familiar adults, relationships and social 
interactions with peers, curiosity and initiative in learning, self-control of feelings and behavior, and engagement 
and persistence. The items within the domain were strongly correlated with one another (estimated internal con- 
sistency of .94), and the reliability was .90. 


Another seven items were found to represent a domain identified as academic knowledge and skills. Those seven 
items were related to comprehension of age-appropriate text, phonological awareness, letter and word knowl- 
edge, classification of objects into groups based on their attributes, number sense of quantity, ability to add and 
subtract small quantities, and knowledge of shapes. The items within the domain were strongly correlated with 
one another (estimated internal consistency of .92), and the reliability was .90. 


The two-domain structure was valid across all child subgroups 
The psychometric analyses indicate that the two-domain structure did not vary across subgroups. The study 
team looked for inconsistency in the factor structure across eligibility for the national school lunch program, 


gender, English learner status, eligibility for an individualized education program, and race/ethnicity (see table C2 
in appendix C) but found none. 
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Nearly 9 in 10 children (88 percent) had a score below the scale’s midpoint for the learning and social 
skills domain, and 85 percent had a score below the scale’s midpoint for the academic knowledge and 
skills domain 


The average score for the learning and social skills domain was 2.5 on a scale of 1 to 6 (table 2), and about 88 
percent of children had a score that was below the scale’s midpoint, 3.5 (see figure C2 in appendix C). The average 
score for the academic knowledge and skills domain was 2.7 (see table 2), and about 85 percent of children had a 
score that was below the scale’s midpoint (see figure C3 in appendix C). 


Table 2. Descriptive statistics of scores for the two domains that the Kindergarten Individual Development 
Survey measures, fall 2017 administration in Illinois 


Standard 
Domain Mean deviation Ayer) (-) 
Learning and social skills 25, 0.98 1-6 
Academic knowledge and skills 2.7 0.98 1-6 


Note: The sample consisted of 113,716 children in 2,039 schools. 


Source: Authors’ analyses of fall 2017 Kindergarten Individual Development Survey data provided by the Illinois State Board of Education. 


Skills at kindergarten entry were higher for some child subgroups than for others 


The results of multilevel modeling indicated that skills at kindergarten entry were higher for some child sub- 
groups than for others. 


Children who were not eligible for the national school lunch program had higher scores for both domains than children 
who were eligible. Children who were not eligible for the national school lunch program had an average score of 
2.6 for the learning and social skills domain, while children who were eligible had an average score of 2.3 (figure 1). 
Children who were not eligible for the national school lunch program had an average score of 2.8 for the academic 
knowledge and skills domain, while children who were eligible had an average score of 2.5. Children who were eligible 
for the national school lunch program had a disadvantage of 0.30 standard deviation for the learning and social skills 
domain and 0.43 standard deviation for the academic knowledge and skills. The gap for the academic knowledge and 
skills domain is equivalent to 51 days of kindergarten instruction, which is considered substantively meaningful. 
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Figure 1. Kindergarteners in Illinois who were not eligible for the national school lunch program had higher 
average scores than kindergarteners who were eligible, for both domains on the Kindergarten Individual 
Development Survey, fall 2017 administration in Illinois 


Learning and social skills domain Academic knowledge and skills 
Sum-score index Sum-score index 

6 6 

5 5 

4— 4 


Overall average 2.7 


Overall average 2.5 


Not eligible for Eligible for Not eligible for Eligible for 


the national the national the national the national 
school lunch program school lunch program school lunch program school lunch program 


Note: The analyses were conducted using multilevel modeling and accounted for clustering of children in classrooms and districts. The sample consisted 
of 61,844 children in 1,844 schools who were not eligible for the national school lunch program and 51,626 children in 1,958 schools who were eligible. 
Data on eligibility for the national school lunch program were not available for 246 children. 


Source: Authors’ analyses of fall 2017 Kindergarten Individual Development Survey data provided by the Illinois State Board of Education. 


Girls had higher scores for both domains than boys. Girls had an average score of 2.6 for the learning and social 
skills domain, while boys had an average score of 2.4 (figure 2). Girls had an average score of 2.7 for the academic 
knowledge and skills domain, while boys had an average score of 2.6. Boys had a disadvantage of 0.17 standard 
deviation for the learning and social skills domain and 0.04 standard deviation for the academic knowledge and 
skills domain. The gap for the academic knowledge and skills domain is equivalent to 5 days of kindergarten 
instruction, which is not considered substantively meaningful. 


Figure 2. Girls in kindergarten in Illinois had higher average scores than boys for both domains on the 
Kindergarten Individual Development Survey, fall 2017 administration in Illinois 


Learning and social skills domain Academic knowledge and skills 
Sum-score index Sum-score index 

6 6 

5 5 

4 4 

3 3 


Overall average 2.5 Overall average 2.7 


Girls Boys Girls Boys 


Note: The analyses were conducted using multilevel modeling and accounted for clustering of children in classrooms and districts. The sample consisted 
of 55,980 girls in 2,029 schools and 57,736 boys in 2,033 schools. Both groups appear to be below the overall average for the academic knowledge and 
skills domain because the scale of the figure limits the presentation of the minute differences involved: the value for girls is 2.67, the value for boys is 
2.63, and the overall average is 2.66. 


Source: Authors’ analyses of fall 2017 Kindergarten Individual Development Survey data provided by the Illinois State Board of Education. 
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Non-English learner children had higher scores for both domains than English learner children. Non—English learner 
children had an average score of 2.5 for the learning and social skills domain, while English learner children had an 
average score of 2.3 (figure 3). Non-English learner children had an average score of 2.7 for the academic knowl- 
edge and skills domain, while English learner children had an average score of 2.5. English learner children had a 
disadvantage of 0.20 standard deviation for the learning and social skills domain and 0.18 standard deviation for 
the academic knowledge and skills domain. The gap for the academic knowledge and skills domain is equivalent 
to 21 days of kindergarten instruction, which is considered substantively meaningful. 


Figure 3. Non-English learner kindergarteners in Illinois had higher average scores than English learner 
kindergarteners for both domains on the Kindergarten Individual Development Survey, fall 2017 
administration in Illinois 


Learning and social skills domain Academic knowledge and skills 
Sum-score index Sum-score index 

6 6 

5 5 

4 4 

3 3 
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learner children children learner children children 


Note: The analyses were conducted using multilevel modeling and accounted for clustering of children in classrooms and districts. The sample consisted 
of 85,784 non-English learner children in 2,026 schools and 27,932 English learner children in 1,405 schools. Both groups appear to be below the overall 
average for the academic knowledge and skills domain because the scale of the figure limits the presentation of the minute differences involved: the 
value for non-English learner children is 2.69, the value for English learner children is 2.52, and the overall average is 2.66. 


Source: Authors’ analyses of fall 2017 Kindergarten Individual Development Survey data provided by the Illinois State Board of Education. 


Children who were not eligible for an individualized education program had higher scores for both domains than 
children who were eligible. Children who were not eligible for an individualized education program had an average 
score of 2.5 for the learning and social skills domain, while children who were eligible had an average score of 2.0 
(figure 4). Children who were not eligible for an individualized education program had an average score of 2.7 for 
the academic knowledge and skills domain, while children who were eligible had an average score of 2.3. Children 
who were eligible for an individualized education program had a disadvantage of 0.51 standard deviation for the 
learning and social skills domain and 0.38 standard deviation for the academic knowledge and skills domain. The 
gap for the academic knowledge and skills domain is equivalent to 45 days of kindergarten instruction, which is 
considered substantively meaningful. 
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Figure 4. Kindergarteners in Illinois who were not eligible for an individualized education program had higher 
average scores than kindergarteners who were eligible, for both domains on the Kindergarten Individual 
Development Survey, fall 2017 administration in Illinois 


Learning and social skills domain Academic knowledge and skills 
Sum-score index Sum-score index 

6 6 

5 5 

4 4 

3 3 


Overall average 2.7 
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education program education program education program education program 


Note: The analyses were conducted using multilevel modeling and accounted for clustering of children in classrooms and districts. The sample consisted 
of 102,775 children in 2,033 schools who were not eligible for an individualized education program and 10,941 children in 1,811 schools who were eligible. 
Both groups appear to be below the overall average for the academic knowledge and skills domain because the scale of the figure limits the presenta- 
tion of the minute differences involved: the value for children not eligible for an individualized education program is 2.68, the value for children eligible 
for an individualized education program is 2.32, and the overall average is 2.66. 


Source: Authors’ analyses of fall 2017 Kindergarten Individual Development Survey data provided by the Illinois State Board of Education. 


White and Asian children had higher average scores for both domains than Hispanic and Black children. White and 
Asian children had an average score of 2.5 for the learning and social skills domain, while Hispanic children and 
Black children had an average score of 2.4 (figure 5). White children had an average score of 2.7 for the academic 
knowledge and skills domain, and Asian children had an average score of 2.9, while Hispanic children and Black 
children had an average score of 2.5. White children had a disadvantage of 0.18 standard deviation relative to 
Asian children for the academic knowledge and skills domain. That gap is equivalent to 22 days of kindergarten 
instruction, which is considered substantively meaningful. Hispanic children had a disadvantage of 0.14 standard 
deviation relative to White children for the learning and social skills domain and 0.24 standard deviation for the 
academic knowledge and skills domain. The gap for the academic knowledge and skills domain is equivalent to 28 
days of kindergarten instruction, which is considered substantively meaningful. Black children had a disadvantage 
of 0.19 standard deviation relative to White children for the learning and social skills domain and 0.20 standard 
deviation for the academic knowledge and skills domain. The gap for the academic knowledge and skills domain is 
equivalent to 24 days of kindergarten instruction, which is considered substantively meaningful. 
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Figure 5. White and Asian kindergarteners in Illinois had higher average scores than Black and Hispanic 
kindergarteners for both domains on the Kindergarten Individual Development Survey, fall 2017 
administration in Illinois 


Learning and social skills domain Academic knowledge and skills 
Sum-score index Sum-score index 
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Note: The analyses were conducted using multilevel modeling and accounted for clustering of children in classrooms and districts. The sample consisted 
of 57,869 White children in 1,740 schools, 6,225 Asian children in 881 schools, 137 Pacific Islander children in 111 schools, 5,011 multiracial children in 
1,320 schools, 301 American Indian children in 228 schools, 26,598 Hispanic children in 1,624 schools, and 17,575 Black children in 1,437 schools. 


Source: Authors’ analyses of fall 2017 Kindergarten Individual Development Survey data provided by the Illinois State Board of Education. 


As the percentage of children in a school who are eligible for the national school lunch program rises, 
child scores for both domains decline 


School-level poverty—the percentage of children in a school who are eligible for the national school lunch 
program—had a statistically significant negative association with child scores for both the learning and social 
skills domain and the academic knowledge and skills domain (see tables C6 and C7 in appendix C). After child-lev- 
el characteristics (including children’s eligibility for the national school lunch program) were controlled for, the 
association between school-level poverty and child scores for the learning and social skills domain was no longer 
statistically significant, but the association between school-level poverty and scores for the academic knowledge 
and skills domain remained statistically significant, though weaker. For example, after child-level eligibility for the 
national school lunch program was controlled for, a child attending a school where 20 percent of children were 
eligible for the national school lunch program had an average score of 2.7 for the academic knowledge and skills 
domain, while a child attending a school where 90 percent of children were eligible had an average score of 2.5, a 
difference of 0.24 standard deviation. The gap for the academic knowledge and skills domain is equivalent to 28 
days of kindergarten instruction, which is considered substantively meaningful. 


Teachers and principals suggested several ways that their schools, districts, and the state can support 
administration of the Kindergarten Individual Development Survey 


Interviews with a sample of one kindergarten teacher and one principal from each of nine schools yielded infor- 
mation about the barriers to using the Kindergarten Individual Development Survey and about suggestions for 
how state and local education agencies could improve future survey administration. 


Teachers and principals requested more information about the intended purposes of the assessment from their 
schools, districts, and the state. Four teachers and principals reported that they were uncertain how to use the 
Kindergarten Individual Development Survey data to inform their instruction or school decisions. Teachers and 
principals alike expressed uncertainty about whether the state would use the data for accountability purposes. 


REL 2020-012 11 


As one principal said, “[Teachers] just felt like they were doing it, and it went into a big black hole, and they really 
didn’t get any feedback or information. They did not feel like it was guiding their instruction.... | think a lot of 
people are doing the bare minimum to be able to check a box.” To increase teachers’ engagement in collecting 
and using data, teachers and principals suggested that the state provide additional information about how the 
data could be used at multiple levels of the school system. 


Teachers and principals reported that competing priorities were a barrier to administering the survey at the begin- 
ning of the school year. At the beginning of the school year, teachers reported that they prioritize establishing 
classroom routines and behavioral expectations. They said that adding the data collection and reporting for the 
Kindergarten Individual Development Survey to beginning-of-the-year duties was difficult, especially if their dis- 
trict also required additional beginning-of-the-year assessments or if it was their first time conducting the survey. 


According to teachers and principals, one way to reduce the burden associated with the survey might be for 
schools or districts to change the requirements surrounding report cards and mandated beginning-of-the-year 
testing. For example, two schools moved kindergarten to a trimester schedule to more easily use the survey 
reports as report cards because KIDStech, the survey’s associated online data system, allows teachers to rate 
and enter data three times a year. The change reduced the additional paperwork that teachers had to complete. 
Schools that eliminated district-mandated assessments that were redundant with the Kindergarten Individual 
Development Survey also lessened the overall burden of testing and assessment for teachers.” 


Several teachers and principals reported that shifting to a play-based or center-based curriculum enabled teachers 
to observe children’s skills and facilitated survey administration. The Kindergarten Individual Development Survey 
requires teachers to observe children as they engage in activities and interact with one another. In classrooms 
where most activities were whole group and teacher led, teachers said that they found it difficult to lead the class 
and observe children simultaneously. For example, one teacher reported that it was challenging to observe chil- 
dren’s social-emotional development and approaches to learning because her classroom did not allow for play or 
child-led activity. Although inconsistent with the intent of the assessment, two teachers chose to assess children 
one on one while other school staff monitored the rest of the class because they found it challenging to observe 
children in the context of normally occurring classroom activity. 


Furthermore, in classes where most activities were teacher led, teachers’ lesson plans for the first 40 days of kin- 
dergarten did not always include activities that pertained to the fundamental skills that the survey measures. For 
example, one teacher reported that in past years she did not teach geometric shapes until the end of the school 
year. To complete the survey, this teacher had to decide whether to introduce the study of shapes earlier than 
planned or report that she was unable to rate the skill. If the teacher had had a better understanding of how to 
incorporate the standard into standalone child-centered activities so she could observe children’s knowledge of 
shapes, she might not have encountered this dilemma. 


To accommodate the need for observation, several schools that participated in the pilot of the Kindergarten Indi- 
vidual Development Survey shifted to a more play-based kindergarten curriculum or purchased new materials for 
child-led centers. For example, one principal worked with a curriculum director to design a play-based curriculum 
to facilitate survey administration through observation and set up summer professional development to help 
teachers learn about and incorporate play-based pedagogy. 


2. Districts may use other assessments to track skill development longitudinally into other grades. For example, it is common to adminis- 
ter a literacy assessment such as the Benchmark Assessment System or the Developmental Reading Assessment multiple times over the 
course of several grades to identify a child’s reading growth. 
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Teachers and principals requested simplified formats for the KIDStech interface and reports. Several teachers 
reported that the KIDStech system was challenging to use and had specific suggestions for improving it. Teachers 
requested that the state separate the 14 required items from the 41 optional items to facilitate data entry and 
that the font size of the text be increased. Furthermore, teachers suggested that the printable reports from KIDS- 
tech allow the inclusion of teacher comments and be redesigned for use with parents. 


Teachers and principals requested several revisions to survey training. One suggestion was to provide training for 
the Kindergarten Individual Development Survey to a wider target group than kindergarten teachers and princi- 
pals. Six of the nine schools that participated in the interviews used paraprofessionals, specialists, AmeriCorps 
volunteers, or the principal to administer the survey or support the teacher in administering the survey. These 
support staff could benefit from a better understanding of how to assist with survey administration. Although it 
may not be feasible for all school staff to travel to training or leave the classroom to receive it, providing oppor- 
tunities for staff to watch training online at their convenience could enable more teachers and support staff to 
receive necessary training. 


Teachers also suggested improvements to the content of the trainings. Some teachers were uncertain about how 
to choose between adjacent rating categories because of ambiguity in the language describing each. Concerned 
that descriptors could lead to differences in how teachers rated similar skillsets, teachers recommended provid- 
ing more detailed descriptions of the developmental skills assessed in each item. These descriptions could help 
teachers understand the differences in the ratings categories more precisely. In addition, teachers and principals 
suggested that the trainings provide examples of specific classroom activities or lesson plans that teachers could 
use when collecting data for each item. 


Teachers also expressed concern about whether they were rating English learner children correctly because 
those children might have the skills of interest but were not able to understand the teachers’ directions. None of 
the teachers or principals interviewed reported using the survey’s alternate items intended for use with English 
learner children. According to teachers and principals, future trainings could cover when and how to use the 
alternate items with English learner children. 


Implications 


The results of this study have several implications for the Illinois State Board of Education and for local education 
agencies around the state as they plan for future survey administrations and reporting. 


First, state and local education agencies in Illinois could use the study’s psychometric findings to label and describe 
the newly identified domains that the survey measures and to re-envision the reporting of survey results. While 
three domains were identified in the pilot administration of the survey to 26,495 students in 2,913 kindergarten 
classrooms in 422 districts, the current study, based on the 2017 administration of the survey to 125,800 children 
in 6,275 classrooms in all 2,077 public kindergartens in all 745 districts in the state, found support for measures 
of two domains. The two domains were assessed to be reliable and valid across all child subgroups studied. One 
potential application of this finding is to transform the current reporting structure that uses three domains to a 
structure that reports scores for two domains. Alternatively, the state might encourage some districts and teach- 
ers to collect data on more than 14 items that would potentially support measurement of more than two skill 
domains. Researchers could then assess whether the additional items reflect another domain before bringing 
such a data collection to scale statewide. 


Second, differences in kindergarten readiness between student subgroups—particularly if sustained—may imply 
a need for policymakers to explore opportunity gaps in early childhood. This study found substantively important 


differences in skills at kindergarten entry between children eligible for the national school lunch program and 
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those not eligible, between English learner children and non-English learner children, between children eligible 
for an individualized education program and those not eligible, and across children of different races/ethnicities. 
These findings are consistent with patterns observed in nationally representative data, such as Early Childhood 
Longitudinal Study Kindergarten Cohort data (Mulligan, Hastedt, & McCarroll, 2012). In addition to differences 
across subgroups, the study found that school-level poverty was associated with children’s skills on both domains, 
with the association between school-level poverty and the academic knowledge and skills domain remaining sta- 
tistically significant and meaningful after child-level characteristics were controlled for. If these patterns hold in 
future years, state and local education agencies could consider further exploring how opportunity gaps in early 
childhood may lead to differences in skills at kindergarten entry and could consider providing targeted support 
and resources to reduce these differences. 


Third, the suggestions from interviews with teachers and principals could improve the process of generating and 
using the Kindergarten Individual Development Survey data. Teachers and principals reported that their primary 
reason for administering the survey was to follow state mandates, and they expressed confusion about its 
purpose. If these opinions are representative of broader teacher and principal opinions in Illinois, state and local 
education agencies could continue to explain potential applications of the assessment data in the classroom and 
within school systems so that educators feel more invested in survey administration. In addition, teachers report- 
ed difficulty in finding time to observe children and difficulty in determining how to choose between adjacent 
rating categories. To the extent that these reports represent challenges for teachers statewide, state and local 
districts could consider providing additional professional development to support the observation and rating of 
children in the context of developmentally appropriate instruction. Additional research on barriers to collecting 
and using data from the Kindergarten Individual Development Survey is needed to learn more about the repre- 
sentativeness of the interview findings. 


Limitations 


The study assessed some aspects of the reliability of the items, including internal consistency. However, it was not 
possible to examine inter-rater reliability or test-retest reliability. Inter-rater reliability measures whether multi- 
ple raters—in this case, teachers—rate the same child similarly. Because children received ratings only from their 
teacher, it was impossible to assess whether teacher ratings represent a “true” skill level or whether they reflect 
errors introduced when the teacher collected information and entered ratings for the child. Teachers’ ratings may 
be influenced by mistakes in using the survey instrument and KIDStech, misunderstandings of the developmental 
continuum underlying the ratings categories, or implicit or explicit biases (Ready & Wright, 2011). For example, 
many teachers in the 2017/18 school year misunderstood when and how to use the two alternate items to assess 
English learner children (see appendix A for more detail). This misuse of the assessment could affect the reliability 
of scores for English learner children, as well as the estimated gaps between English learner children and non— 
English learner children. 


The study was also unable to determine test-retest reliability, which measures whether results could be repro- 
duced at two points over a short period. Children received ratings at only one point in time, so it was not possible 
to examine the reproducibility of results. For these reasons, the results for research questions 1-4 should be 
interpreted with caution. In the future, if multiple data points can be collected for the same children (either by 
multiple teachers or at different times), inter-rater reliability or test-retest reliability could be examined. 


Besides potential limitations related to the reliability of the data, another limitation is that not every school or 
district reported data, and not every teacher reported data for all children. Approximately 97 percent of schools 
and districts reported at least some data, and 95 percent of children had a rating for at least one of the 14 survey 
items, though only children for whom teachers had entered ratings for at least 12 of the 14 items were included in 
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the sample. The study sample thus represents 86 percent of the total 2017/18 Illinois kindergarten cohort. Analyses 
in appendix B show that the analytic sample is similar to the population of the 2017/18 Illinois kindergarten cohort. 


There are also limitations in the qualitative research. Because the interviews were conducted with a sample of one 
kindergarten teacher and one principal from each of nine schools, the interview findings are not representative 
of the opinions and experiences of all kindergarten teachers or principals in Illinois. Additional research is needed 
about barriers that teachers and principals face in collecting data and applying findings from the assessment. 
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